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TITLE OF THE INVENTION 
INFORMATION PROCESSING DEVICE AND METHOD 



BACKGROUND OF THE INVENTION 



Field Qf the Invention 

[0001] The present invention relates to operating a 
virtual object in compounded real space. 
Description of the Related Art 

[0002] With regard to operating a virtual object using an 
operating device capable of six-degree-of -freedom operations, 
there is a method for aiding operations so that the 
operations made by the user are those desired by the user, 
by using limited physical laws. For example, in the event 
of attempting to move a virtual object placed on a plane in 
a certain virtual space to a different position on the same 
plane, instead of moving the virtual object by reflecting 
the six-degree-of -freedom operations input from the user 
without change, the virtual object is moved over the two- 
dimensional plane while imitating gravitational and 
collision effects, which results in operations more 
intuitive to the user. Also, with regard to performing 
rotating operations on the plane, restricting the rotations 
to those on a single axis wherein a normal vector 



perpendicular to the plane serves as the axis realizes the 
operations intended by the user more than simultaneously 
operating parallel movement components and tri- axial 
rotations all at the same time. Now, a plane which provides 
the restrictions on the operations of the virtual object by 
means of a single plane is referred to as a "constraining 
plane", and a collection of constraining planes, configured 
of multiple planes, is referred to as a "constraining shape". 
[0003] Conventionally, the virtual object in the virtual 
space was modeled beforehand and input, and the model was 
used as a constraining shape. In the event of realizing 
this method in virtual space, a virtual object and a 
constraining shape are represented in a single space, so 
setting the constraining shape was simple. On the other 
hand, with realizing a compounded space, precise position of 
the two spaces, i.e., real space and virtual space, is a 
crucial element, requiring accurate numerical expression 
model representations or shape model representations on the 
real space in order to correlate the constraining shape to 
the real space. The reason is that the model to be set to 
the constraining shape cannot be something which the user 
sets by eye estimation but rather must be a model 
corresponding to the shape of the real space. In the event 
that the virtual object is operated using a numerical 
expression model of a plane not matching a plane in the real 
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space, as the constraining plane, spatial non-continuity 
between the virtual space and real space occurs, such as the 
virtual object floating above or moving below the plane in 
the real space. Upon experiencing such non -continuity, the 
5 user will feel that the inability for objects in the real 

space to affect the virtual object is unnatural, and may not 
be able to perform intuitive operations. However, precise 
modeling of constraining shapes in the real space requires 
that margin of error in the input of geometrically accurate 

10 shapes and in the environment to be realized (margin of 

error in sensor measurement, margin of error with regard to 
displacement of sensors, etc.) has to be taken into 
consideration as well, so realization has been difficult. 
[0004] Thus, there has been the need to prepare precise 

15 numerical expression models and shape models beforehand in 

order to realize the method for aiding operations of virtual 
objects by affecting objects in real space, and further, 
this operating method could only be used where models have 
been prepared. 



20 



S UMMARY QF THE INVENTION 



25 



[0005] The present invention has been made in light of 
these problems, and accordingly, is an object of the present 
invention to enable dynamic creating of constraining shapes 
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in a compounded real space, and to enable easy operating of 
virtual objects using constraining shapes even where 
constraining shapes have not been registered beforehand. 
[0006] In order to achieve the above objects, according 
5 to a first aspect of the present invention, an information 

processing device, for aiding operations relating to 
position and orientation of a virtual object positioned in 
three-dimensional space, comprises: image-taking means for 
taking images in real space in order to display the virtual 

10 object in a superimposed manner on real space; synthesizing 

means for synthesizing a virtual object with the taken 
image; operating means for operating the position and 
orientation of the virtual object; and aiding means for 
obtaining a three-dimensional position of real space from 

15 external instructions, and obtaining a constraining shape 

for aiding in operations for the position and orientation of 
the virtual object; wherein the position and orientation of 
the virtual object are operated by instructions from the 
operating means, based on constraint conditions based on the 

20 constraining shape obtained by the aiding means. 

[0007] Further objects, features and advantages of the 
present invention will become apparent from the following 
description of the preferred embodiments with reference to 
the attached drawings. 
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BRIEF DESCRIPTION O F THE PRAWINGS 



[0008] Fig. 1 is a block diagram illustrating the 
schematic configuration of an embodiment applying the 
operation aiding device for virtual objects in compounded 
real space. 

[0009] Fig. 2 is a model diagram illustrating the 
configuration of the device in the first embodiment. 
[0010] Fig. 3 is a block diagram illustrating the 
internal configuration of a constraining shape generating 
unit 1080. 

[0011] Fig. 4 is a block diagram illustrating the 
configuration of an operating aiding unit 1100. 
[0012] Fig. 5 is a flowchart illustrating the procedures 
for processing with the first embodiment. 

[0013] Fig. 6 is a flowchart illustrating the detailed 
processing procedures relating to generating a constraining 
shape in step S5030. 

[0014] Fig. 7 is a flowchart illustrating the detailed 
processing procedures relating to generating a numerical 
expression model of the constraining shape in step S6010. 
[0015] Fig. 8 is a flowchart illustrating the detailed 
processing procedures relating to generating a numerical 
expression model of the constraining shape in step S5060. 
[0016] Fig. 9 is a model diagram illustrating the manner 
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in which a virtual object 2010X intrudes into a constraining 
shape 2000A. 

[0017] Fig. 10 is a model diagram illustrating inputting 
of the constraining shape in the real space. 
5 [0018] Fig. 11 is a flowchart illustrating the detailed 

processing procedures relating to generating a numerical 
expression model for a constraining shape, according to a 
first modification. 

[0019] Fig. 12 is a model diagram illustrating the method 
10 for generating a numerical expression model for a 

constraining shape, according to a first modification. 
[0020] Fig. 13 is a bird's-eye view from the rear left 
side, illustrating the state in which a virtual object is 
being operated. 

15 [0021] Fig. 14 is a model diagram of an MR image, wherein 

an image taken with an image-taking unit 1010 which a user 
is viewing, and a virtual image, are synthesized. 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 

20 

Fiyst Embodiment 
< 1 . Conf iguration> 

[0022] The following is a detailed description of 
preferred embodiments of the present invention, with 
25 reference to the drawings. 
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[0023] Fig. 1 is a block diagram illustrating the 
schematic configuration of a first embodiment of the present 
invention, applying the operation aiding device for virtual 
objects in compounded real space. Fig. 2 is a model diagram 
5 illustrating the configuration of the device in the first 

embodiment . 

[0024] An image-taking unit 1010, an image-taking unit 
position and orientation measurement unit 1020, and a 
display unit 1030, are attached and fixed to an HMD 1000. 

10 As shown in Fig. 2, two each of the image- taking unit 1010 

and display unit 1030 are provided within the HMD, so that 
image- taking of the real space and recognition of the 
compounded real space can be performed in stereo. The user 
can view the stereo compounded real space images displayed 

15 on the display units 1030 (hereafter referred to as "MR 

(mixed-reality) images") with the HMD 1000 mounted on the 
head. The present invention is not restricted to taking and 
displaying stereo images, and is applicable to arrangements 
having at least one image-taking unit and display unit. 

20 Further, the present invention is not restricted to using an 

HMD 1000, and may be applied to any method wherein an image- 
taking unit and image -taking unit position and orientation 
measurement unit 1020 are fixed, and the user can view a 
display unit 1030. 

25 [0025] The image- taking unit position and orientation 



measurement unit 1020 uses a 3D SPACE FASTRAK sensor 
manufactured by Polhemus in the present embodiment, and as 
shown in Fig. 2, measures the image-taking unit position and 
orientation by receiving a magnetic field generated by a 
magnetic field generating unit 2020 disposed in real space 
with the image-taking unit position and orientation 
measurement unit 1020 which is a receiving device fixed to 
the image- taking unit, and sending change in the received 
magnetic field to a sensor controller 2030. With the 
present invention, the image -taking unit position and 
orientation measurement unit is not restricted to a magnetic 
sensor, rather, any arrangement such as image -taking unit 
position and orientation estimation with an optical sensor 
or with image processing and so forth may be applied, as 
long as the position and orientation of the image-taking 
unit can be measured. 

[0026] A taken-image obtaining unit 1040 holds images 
taken by the image-taking unit 1010 as image data. 
[0027] An image- taking unit position and orientation 
estimation unit 1050 performs estimation of the image-taking 
unit position and orientation, based on the measurement 
values obtained from the image- taking unit position and 
orientation measurement unit 1020. Methods which can be 
applied for this estimation include: high-precision 
obtaining using characteristic points which have known 
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positions in three-dimensional space, from the image held in 
the taken-image obtaining unit 1040; estimation of the 
margin of error of the sensor and correction thereby; and so 
forth. 

5 [0028] A constraining shape input unit 1060 obtains the 

three-dimensional position of a constraining plane 2000 in 
real space, and inputs this to a constraining shape 
generating unit 1080. The present embodiment uses the 
stylus of the 3D SPACE FASTRAK sensor manufactured by 

10 Polhemus. This stylus is provided with one button (not 

shown) serving as a user interface for specifying the three- 
dimensional position of objects in the real space, and so 
forth. The present invention is not restricted to such 
three-dimensional pointing devices, rather, any arrangement 

15 may be applied as long as constraining shapes in real space 

can be precisely modeled, such as a method wherein image 
characteristics of real space are taken with a stereo camera 
to obtain the shape of an object. 

[0029] The constraining shape generating unit 1080 
20 synthesizes the input constraining plane with the 

constraining shapes accumulated therein so far, and stores 
this. Fig. 3 is a block diagram illustrating the inner 
configuration of the constraining shape generating unit 1080. 
A constraining plane generating unit 3000 generates planar 
25 numerical expression models or shape models from the three- 
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dimensional position in the real space that has been input, 
thereby forming a constraining plane. The generated 
constraining shape is stored in a constraining shape storage 
unit 3010. 

[0030] A virtual object operating unit 1090 is provided 
for performing six-degree-of -freedom operations of the 
virtual object 2010. With the present embodiment, the 
stylus of the constraining shape input unit 1060 is also 
used. The present invention is not restricted to the method 
sharing a stylus, and instead may comprise another FASTRAK 
sensor and use the stylus thereof. Further, other optical 
sensors or a two-dimensionally operated mouse or the like 
may be used instead of a FASTRAK sensor. 

[0031] A virtual object placement shape storage unit 1070 
stores the shape of the virtual object 2010, and the 
placement thereof in the scene. The stored virtual object 
2010 is transferred to the operating aiding unit 1100, and 
the placement of the virtual object 2010 within the 
operation aiding unit 1100 or the shape thereof is changed 
according to the interference with the constraining shape 
2000 and the operations input by a virtual object operating 
unit 1090. 

[0032] The operation aiding unit 1100 generates operation 
restrictions from the information of the constraining shape 
generating unit 1080 and the virtual object placement shape 



storage unit 1070, and restricts operations being input from 
the virtual object operating unit 1090. Fig. 4 is a block 
diagram illustrating the configuration of the operation 
aiding unit 1100. A distance judging unit 4000 compares the 
distance between the virtual object 2010 to be operated 
which is stored in the virtual object placement shape 
storage unit 1070 and the constraining shape 2000 stored in 
the constraining shape generating unit 1080. An operation 
constraint generating unit 4010 generates operation 
restrictions with regard to the virtual object of interest, 
following the distance obtained by the distance judging unit 
and the constraining shape. An operation deciding unit 4020 
decides the final operations based on the operation input 
from the virtual object operating unit 1090 and the 
operating restrictions generated at the operation constraint 
generating unit 4010, performs operations on the virtual 
object, and stores the resultant position and orientation of 
the virtual object in the virtual object placement shape 
storage unit 1070. 

[0033] The virtual image generating unit 1110 performs 
two-dimensional projection of the virtual object 2010 on a 
display coordinates system, based on the image -taking unit 
position and orientation estimation unit 1050 and the 
virtual object placement shape storage unit 1070, and thus 
generates a virtual image. 



- 12 - 



[0034] The image synthesizing unit 1120 synthesizes the 
images obtained from the taken-image obtaining unit 1040 and 
the virtual image generating unit 1110. 
[0035] The MR image output unit 1130 transfers the 
synthesized image to the display units 1030. 
[0036] An image corresponding to the point -of -view- 
position of the user is displayed on the display units 1030 
of the HMD 1000. The user can sense the compounded real 
space by viewing this image, and also operate virtual 
objects 2010 in the compounded real space at the same time. 
[0037] Further, with the present embodiment, the 
aforementioned taken-image obtaining unit 1040, image-taking 
unit position and orientation estimation unit 1050, virtual 
object placement shape storage unit 1070, constraining shape 
generating unit 1080, operation aiding unit 1100, virtual 
image generating unit 1110, image synthesizing unit 1120, 
and MR image output unit 1130 are stored in a workstation 
2040 shown in Fig. 2. The workstation 2040 has two separate 
video capture devices for capturing images of the image - 
taking unit, and two separate picture output devices, in 
addition to the basic components. However, the present 
invention is not restricted to such a workstation, and a 
computer having the same functions will suffice. 
<2. Procedures for processing> 

[0038] The procedures for processing according to the 
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present embodiment will be described with reference to the 
flowchart in Fig. 5. 

[0039] In step S5000, an image taken with the image- 
taking unit 1010 is input to the taken-image obtaining unit 
5 1040. 

[0040] In step S5010, the measurement values from the 
image -taking unit position and orientation measurement unit 
1020 are input to the image-taking unit position and 
orientation estimation unit 1050, thereby estimating the 
10 position and orientation of the image-taking unit 1010. A 

4 -by- 4 viewing conversion matrix MC for conversion from a 
world coordinates system to an image-taking unit coordinates 
system is generated, based on the estimated position and 
orientation. 

15 [0041] In step S5020, selection of whether a constraining 

shape is to be input or a virtual object is to be operated, 
is determined by button operation of the constraining shape 
input unit 1060 by the user. The present invention is not 
restricted to methods for selecting this operation by button 

20 operation of the stylus, and may be applied to methods 

wherein selection is made from other input devices such as a 
mouse or keyboard or the like. 

[0042] In step S5020, in the event that a mode for 
inputting a constraining shape is selected, the flow 
25 proceeds to step S5030. Or, in the event that a mode for 
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operating a virtual object is selected, the flow proceeds to 
step S5040. 

[0043] In step S5030, the constraining shape input unit 
1060 inputs points of an object in real space, and generates 
5 a constraining shape. Fig. 10 is a model diagram 

illustrating the manner in which a constraining shape in 
real space is input. The details of step S5030 will be 
described later. 

[0044] In step S5040, the virtual objects 2010 is 

10 operated from the virtual object operating unit 1090. The 

change of position and orientation at the virtual object 
operating unit 1090 due to the input of operations is stored. 
[0045] In step S5050, positions between the constraining 
shape 2000 input at the operation aiding unit 1100 and the 

15 virtual object 2010 updated by operations are compared, and 

in the event that the distance therebetween is smaller than 
a predetermined threshold value, the flow proceeds to step 
S5060 since restrictions of the constraining shape apply. 
In the event that the value is greater than the threshold 

20 value, operations can be performed without the restrictions 

of the constraining shape, so the flow proceeds to step 
S5070. Now in a case wherein translation transformation or 
rotation transformation is performed under constraint of a 
constraining plane 2000A (not shown) where the virtual 

25 object 2010 currently is, in the event that the virtual 
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object following conversion overlaps a separate constraining 
plane 2000B (not shown), the restriction applied is that of 
either the center of gravity of the virtual object 2010 or 
the center of gravity of the constraining plane 2000, 
5 whichever is closer distance-wise. 

[0046] In step S5060, the operations of the virtual 
object 2010 are corrected based on the constraining plane 
2000. Details will be described later. 

[0047] In step S5070, the state of the virtual object 
10 within the virtual object placement shape storage unit 1070 

is updated, according to the results of operation of the 
virtual object. 

[0048] In step S5080, the virtual object 2010 of the 
image-taking unit position and orientation estimation unit 

15 1050 is converted by the viewing conversion matrix MC 

obtained in step S5010, and further is subjected to two- 
dimensional projection with a known projection matrix. 
[0049] In step S5090, the virtual image obtained in step 
S5080 is superimposed on the taken image obtained in step 

20 S5000, and displayed on the display units 1030 by the MR 

image output unit 1130. 

<2.1. Details of procedures for constraining shape 
generating process ing> 

[0050] Fig. 6 is a flowchart illustrating the detailed 
25 processing procedures relating to generating a constraining 
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shape in step S5030. In step S6000, whether to generate a 
numerical expression model from input points or to generate 
a shape model, is selected by the user, by operating the 
button on the stylus . The present invention is not 
5 restricted to methods for selecting this operation by button 

operation of the stylus, and may be applied to methods 
wherein selection is made from other input devices such as a 
mouse or keyboard or the like. In the event that the mode 
for generating numerical expression model is selected, the 

10 flow proceeds to step S6010. 

[0051] In step S6010, a numerical expression model is 
generated based on the object data in the real space which 
has been input. Details of the procedures for processing 
will be described later. The parameters of the numerical 

15 expression model generated are added to the constraining 

shape storage unit 3010 in step S6050. 

[0052] In step S6020, points of an object in real space 
(three-dimensional position) to be registered as a 
constraining shape are received from the constraining shape 

20 input unit 1060. The points a020 in Fig. 10 illustrate 

points on an object to become a constraining shape. 
[0053] In step S6030, in the event that the number of 
points input in step S6020 is less than three, the flow 
returns to step S6020, and awaits input of points. In the 

25 event that three points in the real space have been input. 
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the flow proceeds to step S6040. 

[0054] In step S6040, a triangular patch (a triangular 
patch defined by the points a020 in Fig* 10) is generated 
from the three three-dimensional points that have been input. 
5 In the event of inputting a hexagonal pillar shape in real 

space as shown in Fig. 10 as the constraining shape, a 
triangular patch is generated on the surface by inputting 
apexes of the hexagonal pillar shape as shown in Fig. 10. 
At this time, the three input points section the surface of 

10 the hexagonal pillar shape into triangles. Now, an 

arrangement may be made wherein translucent polygons 
corresponding to the virtual points and patches are 
generated and drawn accompanying the input operations for 
the input points and the generated triangular patch, so that 

15 the user can recognize the input of the constraining plane. 

[0055] In step S6050, the generated numerical expression 
model or shape model is stored in the constraining shape 
storage unit 3010. 

<2.2. Details of procedures for numerical expression model 

20 generating processing> 

[0056] Fig. 7 is a flowchart illustrating the detailed 
processing procedures relating to generating a numerical 
expression model of the constraining shape in step S6010. 
With the present embodiment , a method is provided wherein a 

25 planar numerical expression model is generated from four or 
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more points using least -square approximation, as a numerical 
expression model generating model for the constraining shape. 
[0057] First, description will be made regarding a 
constraining plane suitable for a shape model and a 
5 constraining plane suitable for a numerical expression model. 

A shape model is preferably applied to an object containing 
apexes on the constraining plane within a range measurable 
by the constraining shape input unit 1060 which the user has. 
A constraining shape with higher complexity can be generated 

10 by combining constraining planes input this with a shape 

model. On the other hand, a numerical expression model is 
preferably applied to an object which has an overall shape 
greater than the range measurable by the constraining shape 
input unit 1060. This allows a constraining plane to be set 

15 by inputting four or more points on the same plane to be set 

as the constraining plane within a range measurable by the 
constraining shape input unit 1060, even for regions wherein 
the apexes to be input are too distant to be input, such as 
with the ground. In this case, the numerical expression 

20 model of the constraining plane is a model representing an 

infinite plane. 

[0058] Next, the method for generating a numerical 
expression model for an infinite plane with least -square 
approximation will be described. The expression for a plane 
25 used here is ax + by + cz = 1, which can be represented by 
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vector computation as follows, 
[Expression 1] 



[a b c\- 



x 

y 



= 1 



[0059] This Expression (1) holds for points existing on 
the plane to be obtained. Assuming that all points in a 
point group of n points input thus are on the plane, the 
following expression holds. 
[Expression 2 ] 



[a b c]- 



yi y* 



n 



= [l 1 ... 1] 



10 [0059] Now, let us represent this expression as follows. 

[ Expression 3 ] 
p • X = d 

[0060] Obtaining the generalized inverse matrix for X in 
Expression (3) yields p(a, b, c) which are the parameters of 
15 the numerical expression model of the plane. Now, with the 

generalized inverse matrix for X in Expression (3) as X+, 
the following expression holds. 
[Expression 4] 
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X+ = Xt • (X • Xt) - 1 

and thus 
[Expression 5] 

p = d • Xt = d • Xt • (X • Xt) - 1 
5 [0061] The parameters p(a, b, c) of the numerical 

expression model of the plane can be obtained from the input 
point group with this Expression (5). 

[0062] The procedures of the processing for generating a 
numerical expression model will be described with reference 

10 to the flowchart in Fig. 7, based on the above understanding. 

[0063] In step S7000, points on the plane 2000 in the 
real space to be registered as the constraining shape are 
received from the constraining shape input unit 1060. 
Points aOlO in Fig. 10 indicate the points to be input. 

15 [0064] Next, in step S7010, in the event that the number 

of points input in step S7000 is four or more, the flow 
proceeds to step S7020. In the event that the number of 
input points is less than four, the flow returns to step 
S7000 and awaits input of points. 

20 [0065] In step S7020, the flow returns to step S7000 and 

awaits input of points until there is a command to end input 
of the point group from the user. In the event that there 
is an end command, the flow proceeds to step S7030. 
[0066] In step S7030, the input point group is set to the 

25 matrix X in the Expression (5), and the parameters p of the 
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numerical expression model of the plane are obtained from 
the results of calculating Expression (5). Substituting the 
parameters p obtained here into Expression (1) yields the 
numerical expression for the plane. 
5 [0067] The plane defined by the obtained numerical 

expression may be drawn using a translucent virtual image. 
This allows the user to visually recognize the constraining 
shape, and accordingly easily perform operations of the 
virtual object which is to be operated. 

10 [0068] The present invention is not restricted to having 

this numerical expression model for planes, and may be 
applied to methods for numerical expression modeling for 
various shapes. Further, a method wherein one or multiple 
modules are provided for numerical expression modeling, or a 

15 method for converting into numerical expression models from 

input of apexes of the shape model, may be used. 
<2.3. Details of procedures for constraint processing of 
operations of a virtual object> 

[0069] Fig. 8 is a flowchart illustrating detailed 
20 processing procedures for the operations for a constraining 

shape in step S5060. Here, in the event of operating the 
virtual object 2010, the user holds the virtual object 
operating unit 1090 in hand, brings the virtual object 
operating unit 1090 into spatial contact with the virtual 
25 object, and presses a button (not shown) on the virtual 
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object operating unit 1090, whereby the virtual object 2010 
hereafter is subject to six -degree -of -freedom operations for 
the virtual object by synchronizing the amount of change of 
position and orientation of the virtual object 2010 with the 
5 amount of change of the virtual object operating unit 1090. 

Also, releasing the button on the virtual object operating 
unit 1090 disengages this synchronization. In this case, 
the user has the sensation of "grabbing" the virtual object 
with the virtual object operating unit 1090. 

10 [0070] In step S8000, in the event that the virtual 

object 2010 has come into contact or intruded into the 
constraining shape 2000 due to the operations of the virtual 
object performed in step S5040, the flow proceeds to step 
S8010. Otherwise, the flow proceeds to step S8020. 

15 [0071] In step S8010, the position and orientation of the 

virtual object is corrected, based on the position of the 
constraining shape of the virtual object. This correction 
method will be described with reference to the model diagram 
in Fig. 9, illustrating the manner in which the virtual 

20 object 2010X is intruding into the constraining shape 2000. 

[0072] The points 9010A, 9010B, and 9010C in the figure 
illustrates the apexes of the triangular polygon of the 
virtual object 2010X intruding into the constraining shape 
2000. In step S8010, movement is made to the position of 

25 the point 9020 shown here in contact with the constraining 
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plane 2000 and correction is made, by correction conversion 
indicated by 9000A and 9000B. The correction conversion 
9000 disposes the virtual object 2010X before conversion 
such that one or more of the apexes 9020 of the virtual 
5 object come into contact with the constraining shape 2000 at 

the closest distance from the center of gravity of the 
virtual object. 

[0073] First, of the apexes 9010 before conversion, a 
point 9010A which has intruded into the constraining shape 

10 2000 and also which is the farthest from the constraining 

shape 2000 is selected. Next, a point 9020A where a 
straight line, in the opposite direction of the direction in 
which the virtual object 2010 has intruded into the 
constraining plane 2000, which passes through the selected 

15 point 9010A, intersects with the constraining plane 2000, is 

obtained. Next, the virtual object 2010X is subjected to 
parallel movement by the conversion 9000A which moves from 
9010A to 9020A. Next, of the other apexes 9010B 1 and 90100' 
making up the triangular polygon of the virtual object 

20 following movement, the apex 9010C which has distance with 

the constraining plane 2000 that is within the threshold 
value 9040, is selected, and the virtual object 2010Y is 
subjected to rotational conversion 9000B with 9020A as the 
center of rotation thereof, such that 9010C comes into 

25 contact with the constraining plane 2000. In the event that 



- 24 - 



9010B' is also within the threshold value, the virtual 
object 2010Y is subjected to rotational conversion 9000B 
such that both 9010C and 9010B 1 come into contact with the 
constraining plane 2000. The threshold value 9040 may be 
5 registered by the user beforehand, or may be dynamically 

determined by the user according to the situation. The flow 
then proceeds to step S8020 following correction conversion 
so as to come into contact with the constraining plane 2000 
as described above. 

10 [0074] In step S8020, constraint is applied to the 

operations which the virtual object 2010 is subjected to. 
[0075] This restriction only permits rotations with the 
normal of the constraining plane as the axis thereof, and 
parallel movement parallel to the constraining plane, for 

15 the operations on the virtual object. In the event that 

operations are mode on the virtual object having parallel 
movement components t(tx, ty, tz)T not parallel to the 
constraining plane, the effects of constraint are applied by 
deleting components not parallel to the constraining plane. 

20 For example, the translation transformation defined by the 

world coordinates system is converted into a local 
coordinates system where there is an XZ plane on the 
constraining plane by a conversion matrix TWL, and 0 is set 
to the Y component of the converted parallel movement 

25 component. Then, converting the parallel movement component 
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subjected to conversion from the world coordinates system 
into the local coordinates system by an inverse conversion 
matrix (TWL)-l generates the parallel movement components 
t'(t'x, t'y, t'z)T moving over the constraining plane. The 
calculation is shown in Expression (6). 
[Expression 6] 



deciding unit 4020. The operation deciding unit 4020 
provides the same constraint to the virtual object until the 
virtual object intrudes into another constraining plane 2000 
again such that the target constraining plane is changed. 



operations of the virtual object 2010, the user can visually 
feel the touch and weight of the constraining shape with 
regard to operations of the virtual object, and can operate 
the virtual object 2010 intuitively. 
First modification 

[0078] With the numerical expression model generating 
module in step S6010 in the embodiment described above, a 
planar numerical expression model was calculated by 




[0076] 



This constraint is stored within the operation 



[0077] 



Due to such constraints being applied to the 
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inputting points, but the present invention is not 
restricted to this method. Rather, a planar numerical 
expression model may be calculated by inputting two 
directional vectors. Fig. 11 is a flowchart of a numerical 
expression model generating module S6010 using two 
directional vectors, and Fig. 12 is a model diagram 
illustrating the method for generating this numerical 
expression model. This example will be described following 
the flowchart in Fig. 11. 

[0079] First, in step SbOOO, a straight line parallel to 
the constraining plane 2000 is input by the constraining 
shape input unit 1060. Now, the constraining shape input 
unit 1060 must be placed so as to be parallel to the 
constraining plane 2000 on the real object, as shown in Fig. 
12. Defining the attitude of the constraining shape input 
unit 1060 here as cO as shown in Fig. 12, the X-axial 
direction of cO is parallel to the constraining plane 2000. 
[0080] Next, in step SbOlO, the flow returns to step 
SbOOO until two or more direction vectors are input , and 
input processing is continued. Now, the flow proceeds to 
step Sb020 upon inputting the X-axial direction B of the 
attitude cO ' of the constraining shape input unit 1060', as 
shown in Fig. 12. 

[0081] Next, in step Sb020, calculating the cross product 
of the direction A and the direction B yields a normal line 
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of a plane containing the two direction vectors . Since 
these two direction vectors are parallel to the constraining 
plane 2000, the plane wherein A x B is the normal is also 
parallel to the constraining plane 2000. Here, the plane 
having a normal of A x B, passing through a measurement 
point and a point P obtained by drawing a perpendicular line 
to the constraining plane from the measurement point, serves 
as the constraining plane. With the expression of the plane 
as 

[Expression 7] 

ax + by + cz + d = 0, 
the components of the normal vector obtained correspond to 
(a, b, c) , since the normal is A x B. Thus, the constant d 
can be obtained by substituting the values of the components 
of the normal and the point P into Expression (7), and 
accordingly, the numerical expression can be obtained for 
the constraining plane 2000. 
Second Modification 

[0082] A modification of the method for aiding operations 
of a virtual object in compounded real space using a 
constraining plane generated in the above-described 
embodiment will be described. This virtual object operating 
method will be described with reference to the model 
diagrams in Figs. 13 and 14. The configuration of the 
equipment for this operating method example is the same as 
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that described above. Fig. 13 is a bird's-eye view 
illustrating the state in which a virtual object is being 
operated, and Fig. 14 is a diagram of an MR image which a 
user is viewing. Using this method enables the virtual 
5 object operating unit 1090 of the user to intuitively 

operate virtual objects which are farther away than the 
extent of the range where the virtual object operating unit 
1090 can reach. 

[0083] First, the virtual object 2010 is placed on the 

10 constraining plane 2000 set beforehand. The virtual object 

2010 is operated by the virtual object operating unit 1090, 
but there is no need for the virtual object operating unit 
1090 to be in contact with virtual object 2010 in a three- 
dimensional manner; rather, operations for the virtual 

15 object are performed such that the depth- wise direction of 

the MR image which the user is viewing is ignored, and in 
this state, in the event that the virtual object operating 
unit 1090 comes into contact therewith in two-dimensional 
space and the user further presses the button (not shown) 

20 provided on the virtual object operating unit 1090, the two- 

dimensional position d020 of the virtual object operating 
unit 1090 on the screen and the two-dimensional position of 
the virtual object 2010 is synchronized. In the event that 
the user releases the button, the virtual object 2010 is no 

25 longer synchronized with the virtual object operating unit 
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1090, and the parallel movement of the virtual object ends. 
In this case, even in the event that the user performs six- 
degree -of -freedom operations with the virtual object 
operating unit 1090, the operations applied to the virtual 
object 2010 are constraining to parallel movement on the 
constraining plane alone. 

[0084] The two-dimensional position on the MR image is 
necessary for both in order to detect contact of the virtual 
object operating unit 1090 and the virtual object 2010 on 
the MR image. The two-dimensional position (P'x, P'y)T is 
obtained by the three-dimensional position (Px, Py, Pz)T of 
both on a world coordinates system (dOlO and d030 in Fig. 
13) being converted to a camera coordinates system by a 
viewing conversion matrix MC which is a 4 -by- 4 homogenous 
coordinates conversion matrix from the position and 
orientation of the image-taking unit 1010 obtained by the 
image- taking unit position and orientation measurement unit 
1020, and then being subjected to two-dimensional projection 
on an image coordinates system by a projection matrix MP 
which is a known 4 -by- 3 homogenous coordinates conversion 
matrix. Expression (8) shows this calculation. 
[Expression 8] 
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[0085] Further, in order to convert the parallel movement 
in the two-dimensional space into parallel movement in 
three-dimensional space parallel with the constraining plane, 
5 the starting point d020 of the virtual object operating unit 

1090 on the two-dimensional plane and the end point d020' 
are obtained, intersections (d030, D030') between lines 
passing through the projected two-dimensional positions 
(d020, d020') of each from the point of origin d040 on the 

10 camera coordinates system and the constraining plane 2000 on 

the camera coordinates system, are obtained. Next, with the 
amount of parallel movement of a vector heading for the 
starting point d030 to the end point d030 f on the 
constraining plane as d050, this amount of parallel movement 

15 d050 is subjected to coordinates conversion on the world 

coordinates system by the inverse matrix MC-1 of the model 
view matrix. Moving the virtual object using the amount of 
parallel movement obtained on the world coordinates system 
allows the virtual object operating unit 1090 and the 

20 virtual object 2010 to be synchronously operated on the MR 
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image . 

Other embodiments 

[0086] Arrangements wherein software program code for 
realizing the functions of the above -described embodiments 
5 is supplied to a computer in a system or an apparatus 

connected to various devices so as to cause the various 
devices to realize the functions of the embodiments 
described above, and invention is carried out by the 
computer (CPU or MPU) of the system or apparatus operating 
10 the various devices following the stored program, are also 

encompassed by the present invention. 

[0087] In this case, the software program code itself 
realizes the functions of the above -described embodiments, 
and the program code itself, and means for supplying the 

15 program code to the computer, e.g., a storing medium storing 

program the code, comprise the present invention. Examples 
of storing mediums which can be used for storing the program 
code include floppy (R) disks, hard disks, optical disks, 
magneto-optical disks, CD-ROMs, magnetic tape, non-volatile 

20 memory cards, ROM, and so forth. 

[0088] Also, it is needless to say that the present 
invention encompasses cases not only where the computer 
executing the supplied program code realizes the functions 
of the above embodiments, but also where the program code 

25 cooperatively realizes the functions of the above 
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embodiments with the operating system or other applications 
running on the computer. 

[0089] Further, the scope of the present invention also 
encompasses arrangements wherein the supplied program code 
is written to memory provided to function expansion boards 
in the computer or function expansion units connected to the 
computer, following which a CPU or the like provided to the 
function expansion boards or function storing units performs 
all or part of the actual processing based on instructions 
of the program code, so as to realize the functions of the 
above embodiments thereby. 

[0090] While the present invention has been described 
with reference to what are presently considered to be the 
preferred embodiments, it is to be understood that the 
invention is not limited to the disclosed embodiments. On 
the contrary, the invention is intended to cover various 
modifications and equivalent arrangements included within 
the spirit and scope of the appended claims . The scope of 
the following claims is to be accorded the broadest 
interpretation so as to encompass all such modifications and 
equivalent structures and functions . 



